Interactive web crawler

ABSTRACT

The claimed subject matter provides a system or method for web crawling hidden files. An exemplary method comprises loading a web page with a browser agent, and executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values. A list of form controls may be retrieved from the web page using the browser agent, and the controls may be analyzed using a driver component. Form control values may be sent from the driver component to the browser agent, and an event may be submitted to the web page by the browser agent or scripted content may be run to trigger operations on the web page corresponding to the form control values. A URL may be generated for various form control values using a generalizer.

BACKGROUND

Web indexing typically occurs when search engines collect and order datafrom the web to facilitate efficient information retrieval. Through theuse of an index, a search engine may avoid scanning each and everydocument in a corpus and instead rely on the index to fulfill searchqueries. Typically, a web crawler begins the process of web indexing byfetching web pages. There are several types of crawlers, includingstatic crawlers, dynamic crawlers, and interactive crawlers, as furtherdescribed herein.

Traditional link-based crawlers that access web pages through outlinksof seed uniform resource locators (URLs) with static content may notaccess web pages that exist in the deep or hidden Web. The pages thatlie within the hidden Web are accessible only after they are createddynamically as a result of some input to a web page, usually a web userfilling and submitting web forms. There may be few hyperlinks to thepages that are generated as a result of a user filling and submittingweb forms. Further, among pages that are generated as a result of a userfilling and submitting web forms, few of them have hyperlinks pointingto them from general seed URLs. The hidden web may also include pagesthat are accessible only through links produced by scripted content suchas JavaScript, Flash, or AJAX.

In order to index the hidden Web, some search engines introducealgorithms that generate queries for input in forms on a web page. Thequeries may be constructed by analyzing the static content of the webpage and extracting keywords. A common technique is based on termfrequency-inverse document frequency (TFIDF). The queries may be limitedto default values if default values for a particular input or controlexist. In this manner, only a small number of input combinations onnon-scripted forms generate URLs for inclusion into the web index.Moreover, these generated URLs may contain a large number of invalidcombinations, while dependent controls on non-scripted forms anddependencies between various controls may be ignored, leading to a largenumber of invalid web pages. The URLs that are found to be valid may beexcluded based on a lack of distinction or low informativeness.

SUMMARY

The following presents a simplified summary of the innovation in orderto provide a basic understanding of some aspects described herein. Thissummary is not an extensive overview of the claimed subject matter. Itis intended to neither identify key nor critical elements of the claimedsubject matter nor delineate the scope of the subject innovation. Itssole purpose is to present some concepts of the claimed subject matterin a simplified form as a prelude to the more detailed description thatis presented later.

The subject innovation relates to web crawling hidden files. Anexemplary method comprises loading a web page with a browser agent andexecuting dynamic elements hosted on the web page using the browseragent to insert pre-determined values. A list of form controls may beretrieved from the web page using the browser agent, and the controlsmay be analyzed using a driver component. Form control values may besent from the driver component to the browser agent, and an event may besubmitted to the web page by the browser agent or scripted content maybe run to trigger operations on the web page corresponding to the formcontrol values. A URL may be generated for various form control valuesusing a generalizer.

An exemplary system relates to web crawling hidden files. One exemplarysystem comprises a processing unit and a system memory. The systemmemory, which comprises a computer-readable storage medium, stores codeconfigured to direct the processing unit to load a web page with thebrowser agent module. Any dynamic elements hosted on the web page may beexecuted using the browser agent module to insert pre-determined values.A list of form controls may be retrieved from the web page using thebrowser agent module, and the controls may be analyzed using the drivercomponent module. Form control values may be sent from the drivercomponent module to the browser agent module, and a URL may be generatedfor various form control values using the generalizer module.

Another exemplary embodiment of the subject innovation provides one ormore computer-readable storage media that includes code to direct theoperation of a processing unit. The code may direct the processing unitto load a web page with a browser agent and execute any dynamic elementshosted on the web page using the browser agent to insert pre-determinedvalues. A list of form controls may be retrieved from the web page usingthe browser agent, and the controls may be analyzed using a drivercomponent. Form control values may be sent from the driver component tothe browser agent, and a URL may be generated for various form controlvalues using a generalizer.

The following description and the annexed drawings set forth in detailcertain illustrative aspects of the claimed subject matter. Theseaspects are indicative, however, of but a few of the various ways inwhich the principles of the innovation may be employed, and the claimedsubject matter is intended to include all such aspects and theirequivalents. Other advantages and novel features of the claimed subjectmatter will become apparent from the following detailed description ofthe innovation when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a search engine according to the subjectinnovation;

FIG. 2 is a process flow diagram of a method that provides web crawlingof hidden files according to the subject innovation;

FIG. 3 is a block diagram that is useful in explaining web crawling ofhidden files according to the subject innovation;

FIG. 4 is a block diagram that is useful in explaining dependentcontrols according to the subject innovation;

FIG. 5 is a block diagram of an exemplary networking environment whereinaspects of the claimed subject matter can be employed; and

FIG. 6 is a block diagram of an exemplary operating environment that canbe employed in accordance with the claimed subject matter.

DETAILED DESCRIPTION

The claimed subject matter is described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the subject innovation. It may be evident, however,that the claimed subject matter may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to facilitate describing the subjectinnovation.

As utilized herein, terms “component,” “system,” and the like areintended to refer to a computer-related entity, either hardware,software (e.g., in execution), and/or firmware. For example, a componentcan be a process running on a processor, an object, an executable, aprogram, a function, a library, a subroutine, and/or a computer or acombination of software and hardware. By way of illustration, both anapplication running on a server and the server can be a component. Oneor more components can reside within a process and a component can belocalized on one computer and/or distributed between two or morecomputers. The term “processor” is generally understood to refer to ahardware component, such as a processing unit of a computer system.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anynon-transitory computer-readable device, or media, such as acomputer-readable storage media.

Non-transitory computer-readable storage media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk, andmagnetic strips, among others), optical disks (e.g., compact disk (CD),and digital versatile disk (DVD), among others), smart cards, and flashmemory devices (e.g., card, stick, and key drive, among others). Incontrast, computer-readable media generally (i.e., not necessarilystorage media) may additionally include communication media such astransmission media for electrical or electronic signals and the like.

Those skilled in the art will recognize many modifications may be madeto this configuration without departing from the scope or spirit of theclaimed subject matter. Moreover, the word “exemplary” is used herein tomean serving as an example, instance, or illustration. Any aspect ordesign described herein as “exemplary” is not necessarily to beconstrued as preferred or advantageous over other aspects or designs.

The subject innovation relates to web crawling hidden files. The browseragent may be used to interactively fill in forms for retrieval of webcontent within the hidden Web. The use of a knowledge base to prioritizethe crawling of hidden web content allows for intelligent web crawling.Additionally, the subject innovation may estimate the potential gain ofeach form control value setting under real world constraints.

FIG. 1 is a block diagram of a search engine 100 that provides webcrawling of hidden files according to the subject innovation. The searchengine 100 may use a crawler 102 to fetch web pages. The crawler maycontain a discovery portion used to discover URLs, and a fetchingportion used to fetch URLs. Crawlers may generally be classified intothree categories: static, dynamic, and interactive.

A traditional, static crawler may extract or compute a possible URL forthe web page based on static hypertext markup language (HTML) analysis,fetch content for a given URL, and store the content from the URL. Thestatic crawler may extract an outgoing URL on the HTML source of the webpage. When computing a URL, the static crawler may compute a possibleURL by concatenating several query parameters as indicated on the staticHTML source of the web page. However, query parameters may contain emptyvalues, and web masters may include fake information within values ofquery parameters in static HTML source of the web page. Thus, thecomputed URLs may be unreliable. A dynamic crawler may compute apossible URL for a web page, fetch static content for the URL, andexecute various dynamic components of the web page one time. Each formcontrol on the web page may be loaded with a default value, if possible,and content rendered from the computed URL may be stored along with thedynamically loaded values. With a dynamic crawler, forms may not beidentified and dependent controls may not be loaded. However, aninteractive crawler may identify forms on the web page. An interactivecrawler may fetch the static content of a URL, execute various dynamiccomponents of the web page one time, identify any forms to be filled,and fill the form controls. After one form control is filled, variousdynamic components may be executed interactively based on a driver, andrendered content from the URL may be stored.

The search engine may perform document processing 104 by processingcontent rendered by the crawler 102 in order to prepare the content forindexing. Processing the content may include normalizing variousdocument formats and forms within the rendered content. Additionally,the search engine may perform index generation 106 by generating anindex based on the rendered content. Query processing 108 may beperformed by the search engine in order to process a query to the searchengine. Processing the query may include defining misspelled or vaguequeries. The search engine may perform index serving 110 by gatheringall rendered content related to the query. Matching and ranking 112 maybe performed on the gathered content. A user interface 114 may be usedto display the matched and ranked content from the search engine. Theuser interface 114 may also be used to input queries into the searchengine 100.

FIG. 2 is a process flow diagram of a method 200 that provides webcrawling of hidden files according to the subject innovation. At block202, a web page is loaded. The web page may be loaded a browser agent.At block 204, dynamic elements on the web page are executed. Theexecution occurs similar to a human being browsing the web page andselecting values for various forms and dynamic elements on the page. Inthis manner, forms that do not exist on the static HTML may be revealed.Further, the dynamic elements on the web page may be executed using abrowser agent to insert pre-determined values into the dynamic elements.The pre-determined values may be provided by the knowledge base asdescribed herein.

At block 206, a list of form controls is retrieved. The list of formcontrols may be retrieved from the web page using the browser agent, andmay include the preset values, if any, of each form control as well as alist of candidate values for each form control, if available.Additionally, candidate values for each form control that does not existon the static HTML may be revealed.

At block 208, the form controls are analyzed. The form controls may beanalyzed using a driver component, which may include a utility estimatorthat estimates the expected utility gain for each value setting anddetermines whether the parameter filling should be stopped. Generally,the utility estimator estimates the expected utility gain for each valuesetting based on knowledge base and site-dependent features. The utilityestimator may also estimate the expected utility gain of both a singlevalue and a combination of values, and may be dynamically adapted usingfeedback knowledge. All languages and all domains may be used with theutility estimator, and the utility estimator may also consider historydata, such as previous URLs generated from the form in previousexecutions.

The analysis performed by the driver component may include informationabout the form controls, including their positions, preset values, setvalues, and candidate values. Additionally, analysis of the formcontrols includes making a decision as to what value to set for the formcontrols. During the decision making process, the driver component mayresort to a knowledge base to retrieve relevant knowledge to direct itsdecision making. The knowledge base can contain N-grams built fromsearch query logs, which may be used to prioritize which value to setfor each form control. The knowledge base may also include othercomponents used to direct its decision making. Through the use of theknowledge base, the driver component may consider real world constraintsthat are imposed by real world practice, such as the maximum number ofURLs able to be generated for each site or maximum bandwidthlimitations.

At block 210, form control values may be sent from the driver componentto the browser agent. At block 212, an event is submitted to the webpage. The even may be submitted by the browser agent to triggeroperations on the web page corresponding to the form control values.Scripted content, such as JavaScript, Flash, or AJAX, may be executedlocally within the web browser to trigger an event. Once the operationson the web page are complete, the browser agent may re-fetch the entirecontent of the web page, including the form control list and values fordependent form controls that were previously inactive. The operationsmay include refreshing the form control list or populating the candidatevalues for form controls that are previously not applicable. Processflow may then return to block 204 and execute any dynamic elements onthe webpage, which may include the dependent form controls that werepreviously inactive. Blocks 204-212 may be repeated until someconstraints have been met, such as the real world constraints discussedherein or until all form controls have been executed. In this manner,form controls may be interactively filled and executed using values forthe form controls that have been pre-determined by a component such asthe knowledge base.

At block 214, a URL is generated for various form control values using ageneralizer. The generalizer may also provide a seed URL for a crawler.The driver component may output the URL to an output file, or streamlinethe URL to a downstream component. At the same time, the generalizer mayattempt to generate a URL pattern based the URLs previously generated.In addition to directly generating the URLs within constraintsrecognized by the driver component, the generalizer may use sample URLsto generate other valid URLs through generalizations made from analyzingthe samples. The generalizer may also send user feedback to theknowledge base for use in subsequent iterations. In this manner, otherURLs may be iteratively generated until all controls and their dependentcontrols have been analyzed, or until some other constraint has beenachieved, such as a certain number of URLs have been obtained, or untilall resources have been used, such as time.

FIG. 3 is a block diagram 300 that is useful in explaining web crawlinghidden files according to the subject innovation. A browser agent may beused to load a web page 302 to analyze both static and dynamic forms onthe web page. The subject web page may be found by various methods. Inone scenario, an end-user on the Web could submit the web page forcrawling after noticing the web page does not occur in search results orafter noticing the web page contains forms. Similarly, an automaticanalysis tool may analyze a web search log and to find any newparameterized websites that are missing from a particular web index. Theautomatic analysis tool may go through a web browser history log andfind web pages that contain forms for crawling. Additionally, theautomatic analysis tool may identify hosts that have low coverage usingtraditional crawler, and deep crawl those hosts to find pages containingforms.

The form 304 may be hosted by a web page 302 that allows visitors to thepage to search for cars. When form 304 is initially loaded by thebrowser agent, form control 306 may be the only form control that isactive, or able to be clicked, selected, or changed. In FIG. 3, formcontrol 306 allows a user to select a value for the make of a car. Thebrowser agent can retrieve a list of the form controls from the webpage, and in this scenario, the browser agent will retrieve a list ofthe makes of car available.

A driver component 308 then analyzes the makes of car available.Additionally, the driver component 308 can decide what value to set forthe controls. Data going into the driver component 308 from the website302 may include actual values and candidate values from the forms on aweb page. Data going back to the website 302 from the driver component308 may include values to input into the forms of the web page.

During the decision making process, the driver component may use aknowledge base 310 to retrieve relevant knowledge to direct its decisionmaking. For example, the knowledge base can use search query logs toinitially select the most popular make of car in order to optimize theselection of form control values. The driver component 308 may alsoconsider constraints 312 that are imposed by real world practice. Forexample, in this scenario, the web site that hosts web page 302 mayimpose limitations on the number of times the browser agent can accessthe web page 302. Similarly, the web site may have maximum bandwidthlimitations. The utility estimator can make optimal decisions based onconstraints imported. In the event there are no constraints imported,the utility estimator may be used to generate popular content. Considerthe scenario where one-hundred URLs are to be generated for form 304,while the model form control 314, the maximum price form control 316,and the search within form control 318 are fixed at some value. Utilityestimator can make optimal decisions based on imported constraints fromthe model form control 314, the maximum price form control 316, and thesearch within form control 318.

Once the driver component 308 has analyzed the controls, the formcontrol values may be sent from the driver component 308 to the browseragent. The browser agent can submit an event to the web page 302 or runany embedded scripted content to trigger operations on the web pagecorresponding to the form control values received from the drivercomponent 308. The operations on the web page may reveal additionaldependent controls for which the driver component can re-fetch values,such as the model form control 314. The model form control 314corresponds to selecting a model of car after a particular make has beenselected. This process may be iteratively repeated until all controlsand their dependent controls have been analyzed, or until all resourceshave been used.

The web site 302 could have been created using an individual “surface”page for each combination of controls. A traditional crawler would beable to access these surface pages, but webmasters create very few linksto them. Much of the website's content is hidden by dynamic formcontrols, and traditional crawler would not access this content. As aresult, a traditional crawler may cover a small amount of web pages of awebsite that uses dynamic form controls. For example, at a website foran online retailer, there are typically thousands of products, such thatthe retailer may advertise a few products by putting their links on thehomepage, with the majority of products having no links pointing tothem. The web pages for the majority of products are not accessed by atraditional crawler.

For each form control value, a URL may be sent to the output URL list320 from the driver component 308. The output URLs can be sent to ageneralizer 322 or passed to a downstream component where the content ofthe URLs may be fetched or serve as seed URLs to a crawler. Thegeneralizer 322 may be used to perform incremental indexing or provideseed URLs by generating other valid URLs through generalizations fromthe sample URLs from the output URL list 320. The discovery portion 324of the crawler may include the driver component 308, the knowledge base310, constraints 312, the output URL list 320, and the generalizer 322.

The generalizer has a number of different uses. In the event that a formcontrol has an infinite number of values, such as a date selection, aparameterized form of the corresponding URL may be indexed. Thegeneralizer may generate the parameterized URL, and web pages may becrawled with placeholders for the infinite values. A value may besubmitted to the web page that mimics how an end user of the Web mightinteract with the page. Once the value is submitted for the infinite,parameterized values, the generalizer may substitute in the definitevalues to generate output. In the manner, even controls with an infinitenumber of form control values may be crawled. Additionally, if aparticular URL generated by the generalizer is not being clicked on by alarge number of end users, the generalizer may send feedback to theknowledge base about the low popularity of the URL. The driver componentmay then use feedback from knowledge base and penalize those URLsexpected utility. Additionally, there may also be positive feedback forURL patterns that are clicked often, so that utility estimator mayprioritize those URLs.

The driver component is intelligent in that it selects form controlvalues in the manner of a typical human user. Additionally, the drivercomponent may select values according to its interaction with web sites.Another web site 326 may include a form 328 with a drop down control 330and a textbox 332 in which a user of the Web could type in keywords. Thedriver component 308 may populate the textbox 332 with keywords that itselects according to its previous interaction with the website 326. Forexample, if the website 326 is an online retailer with a large selectionof books, and the driver component implements a search for “book” at thewebsite 326, the search may return a large number of results. However,if the driver component implements a search for “insert” at the website, there may be few results. Accordingly, the driver component canuse this information to select keywords to be used in textbox 332.Further, the driver component could also have intelligence to adaptivelyselect values, and the utility estimator may diversify results based onthe web log history and the current status of the web site.

Using the knowledge base, the form may first populate with form controlvalues that are popular in the human mind. In FIG. 3, the most popularmake may be first selected, followed by the most popular model, the mostpopular price range, and so forth. Once the initial set of queries iscomplete, the second most popular form control values within the initialmost popular make may be analyzed. As described herein, popularity maybe obtained from the knowledge base. Various sources may be included inthe knowledge base, including but not limited to, web search logs,knowledge from the internet, certain ontology, and information providedby third parties, such as market researchers. Accordingly, the formcontrol list may be prioritized using the knowledge base and used tofill in forms found by the browser agent.

FIG. 4 is a block diagram 400 of a web page that is useful in explainingdependent controls according to the subject innovation. The web page 402contains controls to select a year form control 404, a make form control406, and a model form control 408 of a car. The make form control 406 isdependent on the year form control 404. In other words, the make formcontrol 406 cannot be selected until the year form control 404 isselected. Similarly, the model form control 408 is dependent on the makeform control 406, and the model form control 408 cannot be selecteduntil the make form control 406 has been selected. Traditional crawlersmay be unable to recognize the make form control 406 and the model formcontrol 408. Further, even if traditional crawlers could recognize themake form control 406 and the model form control 408, they may generatea large number of invalid combinations, such as the value “1900” for theyear form control 404, or a combination of the value “Honda” for themake 406 with the value “mustang” for the model form control 408.Moreover, if the form is executed using scripted content, traditionalcrawlers, such as static crawlers, may be entirely unable to recognizethe form.

In order to provide additional context for implementing various aspectsof the claimed subject matter, FIGS. 5-6 and the following discussionare intended to provide a brief, general description of a suitablecomputing environment in which the various aspects of the subjectinnovation may be implemented. For example, a method for web crawlinghidden files, as described in FIG. 2, can be implemented in such acomputing environment. While the claimed subject matter has beendescribed above in the general context of computer-executableinstructions of a computer program that runs on a local computer and/orremote computer, those skilled in the art will recognize that thesubject innovation also may be implemented in combination with otherprogram modules. Generally, program modules include routines, programs,components, data structures, etc., that perform particular tasks and/orimplement particular abstract data types.

Moreover, those skilled in the art will appreciate that the subjectinnovation may be practiced with other computer system configurations,including single-processor or multi-processor computer systems,minicomputers, mainframe computers, as well as personal computers,hand-held computing devices, microprocessor-based and/or programmableconsumer electronics, and the like, each of which may operativelycommunicate with one or more associated devices. The illustrated aspectsof the claimed subject matter may also be practiced in distributedcomputing environments where certain tasks are performed by remoteprocessing devices that are linked through a communications network.However, some, if not all, aspects of the subject innovation may bepracticed on stand-alone computers. In a distributed computingenvironment, program modules may be located in local and/or remotememory storage devices.

FIG. 5 is a schematic block diagram of a sample-computing system 500with which web crawling hidden files can be implemented. The system 500includes one or more client(s) 510. The client(s) 510 can be hardwareand/or software (e.g., threads, processes, computing devices). Thesystem 500 also includes one or more server(s) 520. The server(s) 520can be hardware and/or software (e.g., threads, processes, computingdevices).

One possible communication between a client 510 and a server 520 can bein the form of an event submitted to a web page residing on server 520.The event may trigger other controls to become active on the web page.The system 500 includes a communication framework 540 that can beemployed to facilitate communications between the client(s) 510 and theserver(s) 520. The client(s) 510 are operably connected to one or moreclient data store(s) 550 that can be employed to store information localto the client(s) 510. Information in the data store(s) 550 may includethe web index resulting from web crawling hidden files. The client datastore(s) 550 do not have to be in the client(s) 510, but may be locatedremotely, such as in a cloud server. Similarly, the server(s) 520 areoperably connected to one or more server data store(s) 530 that can beemployed to store information local to the servers 520. As an example,the client(s) 510 may be computers providing access to the Internet overa communication framework 540. The server(s) 520 may host websitesaccessed by the client(s) 510.

With reference to FIG. 6, an exemplary environment 600 for implementingvarious aspects of the claimed subject matter includes a computer 602.The computer 602 includes a processing unit 604, a system memory 606,and a system bus 608. The system bus 608 couples system componentsincluding, but not limited to, the system memory 606 to the processingunit 604. The processing unit 604 can be any of various availableprocessors. Dual microprocessors and other multiprocessor architecturesalso can be employed as the processing unit 604. The system bus 608 canbe any of several types of bus structure(s) including the memory bus ormemory controller, a peripheral bus or external bus, and/or a local bususing any variety of available bus architectures known to those ofordinary skill in the art.

The system memory 606 may include non-transitory computer-readablestorage media comprising volatile memory 610 and nonvolatile memory 612.The basic input/output system (BIOS), containing the basic routines totransfer information between elements within the computer 602, such asduring start-up, is stored in nonvolatile memory 612. By way ofillustration, and not limitation, nonvolatile memory 612 can includeread only memory (ROM), programmable ROM (PROM), electricallyprogrammable ROM (EPROM), electrically erasable programmable ROM(EEPROM), or flash memory.

Volatile memory 610 includes random access memory (RAM), which acts asexternal cache memory. By way of illustration and not limitation, RAM isavailable in many forms such as static RAM (SRAM), dynamic RAM (DRAM),synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhancedSDRAM (ESDRAM), SynchLink™ DRAM (SLDRAM), Rambus® direct RAM (RDRAM),direct Rambus® dynamic RAM (DRDRAM), and Rambus® dynamic RAM (RDRAM).

The computer 602 also includes other non-transitory computer-readablemedia, such as removable/non-removable, volatile/non-volatile computerstorage media. FIG. 6 shows, for example a disk storage 614. Diskstorage 614 includes, but is not limited to, devices like a magneticdisk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-100drive, flash memory card, or memory stick.

In addition, disk storage 614 can include storage media separately or incombination with other storage media including, but not limited to, anoptical disk drive such as a compact disk ROM device (CD-ROM), CDrecordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or adigital versatile disk ROM drive (DVD-ROM). To facilitate connection ofthe disk storage devices 614 to the system bus 608, a removable ornon-removable interface is typically used such as interface 616.

It is to be appreciated that FIG. 6 describes software that acts as anintermediary between users and the basic computer resources described inthe suitable operating environment 600. Such software includes anoperating system 618. Operating system 618, which can be stored on diskstorage 614, acts to control and allocate resources of the computer 602.

System applications 620 take advantage of the management of resources byoperating system 618 through program modules 622 and program data 624stored either in system memory 606 or on disk storage 614. It is to beappreciated that web crawling hidden files can be implemented withvarious operating systems or combinations of operating systems. Thebrowser in which the browser agent operates may be a program module 622.

A user enters commands or information into the computer 602 throughinput device(s) 626. Input devices 626 include, but are not limited to,a pointing device (such as a mouse, trackball, stylus, or the like), akeyboard, a microphone, a joystick, a satellite dish, a scanner, a TVtuner card, a digital camera, a digital video camera, a web camera,and/or the like. The input devices 626 connect to the processing unit604 through the system bus 608 via interface port(s) 628. Interfaceport(s) 628 include, for example, a serial port, a parallel port, a gameport, and a universal serial bus (USB).

Output device(s) 630 use some of the same type of ports as inputdevice(s) 626. Thus, for example, a USB port may be used to provideinput to the computer 602 and to output information from computer 602 toan output device 630. Information rendered by the subject innovation mayappear on an output device 630.

Output adapter 632 is provided to illustrate that there are some outputdevices 630 like monitors, speakers, and printers, among other outputdevices 630, which are accessible via adapters. The output adapters 632include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device 630and the system bus 608. It can be noted that other devices and/orsystems of devices provide both input and output capabilities such asremote computer(s) 634.

The computer 602 can be a server hosting a website in a networkedenvironment using logical connections to one or more remote computers,such as remote computer(s) 634. The remote computer(s) 634 may be clientsystems configured with web browsers, PC applications, mobile phoneapplications, and the like, to allow users to browse the Internet, asdiscussed herein. The remote computer(s) 634 can be a personal computer,a server, a router, a network PC, a workstation, a microprocessor basedappliance, a mobile phone, a peer device or other common network nodeand the like, and typically includes many or all of the elementsdescribed relative to the computer 602. For purposes of brevity, only amemory storage device 636 is illustrated with remote computer(s) 634.Remote computer(s) 634 is logically connected to the computer 602through a network interface 638 and then physically connected via acommunication connection 640.

Network interface 638 encompasses wire and/or wireless communicationnetworks such as local-area networks (LAN) and wide-area networks (WAN).LAN technologies include Fiber Distributed Data Interface (FDDI), CopperDistributed Data Interface (CDDI), Ethernet, Token Ring and the like.WAN technologies include, but are not limited to, point-to-point links,circuit switching networks like Integrated Services Digital Networks(ISDN) and variations thereon, packet switching networks, and DigitalSubscriber Lines (DSL).

Communication connection(s) 640 refers to the hardware/software employedto connect the network interface 638 to the bus 608. While communicationconnection 640 is shown for illustrative clarity inside computer 602, itcan also be external to the computer 602. The hardware/software forconnection to the network interface 638 may include, for exemplarypurposes only, internal and external technologies such as, mobile phoneswitches, modems including regular telephone grade modems, cable modemsand DSL modems, ISDN adapters, and Ethernet cards.

An exemplary embodiment of the computer 602 may comprise a clientcomputer with a web browser. An exemplary processing unit 604 for theclient may be a computing cluster comprising Intel® Xeon CPUs. The diskstorage 614 may comprise an enterprise data storage system, for example,holding thousands of user pages.

What has been described above includes examples of the subjectinnovation. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe claimed subject matter, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the subjectinnovation are possible. Accordingly, the claimed subject matter isintended to embrace all such alterations, modifications, and variationsthat fall within the spirit and scope of the appended claims.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms (including a reference to a “means”) used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., a functional equivalent), even though not structurallyequivalent to the disclosed structure, which performs the function inthe herein illustrated exemplary aspects of the claimed subject matter.In this regard, it will also be recognized that the innovation includesa system as well as a computer-readable storage media havingcomputer-executable instructions for performing the acts and/or eventsof the various methods of the claimed subject matter.

There are multiple ways of implementing the subject innovation, e.g., anappropriate API, tool kit, driver code, operating system, control,standalone or downloadable software object, etc., which enablesapplications and services to use the techniques described herein. Theclaimed subject matter contemplates the use from the standpoint of anAPI (or other software object), as well as from a software or hardwareobject that operates according to the techniques set forth herein. Thus,various implementations of the subject innovation described herein mayhave aspects that are wholly in hardware, partly in hardware and partlyin software, as well as in software.

The aforementioned systems have been described with respect tointeraction between several components. It can be appreciated that suchsystems and components can include those components or specifiedsub-components, some of the specified components or sub-components,and/or additional components, and according to various permutations andcombinations of the foregoing. Sub-components can also be implemented ascomponents communicatively coupled to other components rather thanincluded within parent components (hierarchical). Additionally, it canbe noted that one or more components may be combined into a singlecomponent providing aggregate functionality or divided into severalseparate sub-components, and any one or more middle layers, such as amanagement layer, may be provided to communicatively couple to suchsub-components in order to provide integrated functionality. Anycomponents described herein may also interact with one or more othercomponents not specifically described herein but generally known bythose of skill in the art.

In addition, while a particular feature of the subject innovation mayhave been disclosed with respect to only one of several implementations,such feature may be combined with one or more other features of theother implementations as may be desired and advantageous for any givenor particular application. Furthermore, to the extent that the terms“includes,” “including,” “has,” “contains,” variants thereof, and othersimilar words are used in either the detailed description or the claims,these terms are intended to be inclusive in a manner similar to the term“comprising” as an open transition word without precluding anyadditional or other elements.

1. A method of web crawling hidden files, comprising: loading a web pagewith a browser agent; executing any dynamic elements hosted on the webpage using the browser agent to insert pre-determined values; retrievinga list of form controls from the web page using the browser agent;analyzing the controls using a driver component; sending form controlvalues from the driver component to the browser agent; submitting anevent to the web page by the browser agent or running any scriptedcontent to trigger operations on the web page corresponding to the formcontrol values; and generating a URL for various form control valuesusing a generalizer.
 2. The method recited in claim 1, comprisingre-fetching new form control lists and corresponding values for anydependent controls or web page content using the browser agent.
 3. Themethod recited in claim 1, wherein the driver component uses a knowledgebase that provides knowledge relevant to the driver component's decisionmaking.
 4. The method recited in claim 1, wherein the driver componentincorporates real world constraints into decision making.
 5. The methodrecited in claim 1, comprising a utility estimator to estimate thepotential gain of each form control value setting under real worldconstraints.
 6. The method recited in claim 1, wherein the generalizerprovides a seed URL for a crawler, or the generalizer generates anothervalid URL based on generalizations or incremental indexing.
 7. Themethod recited in claim 1, comprising iteratively generating another URLuntil all controls and their dependent controls have been analyzed,until a certain number of URLs have been obtained, or until allresources have been used.
 8. A system for web crawling hidden files, thesystem comprising: a processing unit; and a system memory, wherein thesystem memory comprises code configured to direct the processing unitto: load a web page via a browser agent module; execute any dynamicelements hosted on the web page using the browser agent module to insertpre-determined values; retrieve a list of form controls from the webpage using the browser agent module; analyze the controls using a drivercomponent module; send form control values from the driver componentmodule to the browser agent module; and generate a URL for various formcontrol values using a generalize module.
 9. The system recited in claim8, comprising re-fetching new form control lists and correspondingvalues for any dependent controls or web page content using the browseragent module.
 10. The system recited in claim 8, wherein the drivercomponent module uses a knowledge base that provides knowledge relevantto the driver component module's decision making.
 11. The system recitedin claim 8, wherein the driver component module incorporates real worldconstraints into decision making.
 12. The system recited in claim 8,comprising a utility estimator to estimate the potential gain of eachform control value setting under real world constraints.
 13. The systemrecited in claim 8, generalizer module provides a seed URL for acrawler, or the generalize module generates another valid URL based ongeneralizations or incremental indexing.
 14. The system recited in claim8, comprising iteratively generating another URL until all controls andtheir dependent controls have been analyzed, until a certain number ofURLs have been obtained, or until all resources have been used.
 15. Oneor more computer-readable storage media, comprising code configured todirect a processing unit to: load a web page with a browser agent;execute any forms hosted on the web page using the browser agent toinsert pre-determined values; retrieve a list of form controls from theweb page using the browser agent; analyze the controls using a drivercomponent; send form control values from the driver component to thebrowser agent; and generate a URL for various form control values usinga generalizer.
 16. The one or more computer-readable storage mediarecited in claim 15, comprising re-fetching new form control lists andcorresponding values for any dependent controls and web page contentusing the browser agent.
 17. The one or more computer-readable storagemedia recited in claim 15, wherein the driver component uses a knowledgebase that provides knowledge relevant to the driver component's decisionmaking, or the driver component incorporates real world constraints intodecision making.
 18. The one or more computer-readable storage mediarecited in claim 15, comprising a utility estimator to estimate thepotential gain of each form control value setting under real worldconstraints.
 19. The one or more computer-readable storage media recitedin claim 15, the generalizer provides a seed URL for a crawler or thegeneralizer generates another valid URL based on generalizations orincremental indexing.
 20. The one or more computer-readable storagemedia recited in claim 15, comprising iteratively generating another URLuntil all controls and their dependent controls have been analyzed,until a certain number of URLs have been obtained, or until allresources have been used.